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We have developed and tested an advanced EVA communications and computing system 
to increase astronaut self-reliance and safety, reducing dependence on continuous 
monitoring and advising from mission control on Earth. This system, called Mobile Agents 
(MA), is voice controlled and provides information verbally to the astronauts through 
programs called “personal agents.” The system partly automates the role of CapCom in 
Apollo — including monitoring and managing EVA navigation, scheduling, equipment 
deployment, telemetry, health tracking, and scientific data collection. EVA data are stored 
automatically in a shared database in the habitat/vehicle and mirrored to a site accessible by 
a remote science team. The program has been developed iteratively in the context of use, 
including six years of ethnographic observation of field geology. Our approach is to develop 
automation that supports the human work practices, allowing people to do what they do 
well, and to work in ways they are most familiar. Field experiments in Utah have enabled 
empirically discovering requirements and testing alternative technologies and protocols. 
This paper reports on the 2004 system configuration, experiments, and results, in which an 
EVA robotic assistant (ERA) followed geologists approximately 150 m through a winding, 
narrow canyon. On voice command, the ERA took photographs and panoramas and was 
directed to move and wait in various locations to serve as a relay on the wireless network. 
The MA system is applicable to many space work situations that involve creating and 
navigating from maps (including configuring equipment for local topology), interacting with 
piloted and unpiloted rovers, adapting to environmental conditions, and remote team 
collaboration involving people and robots. 


I. Introduction 

E XTRAVEHICULAR ACTIVITIES (EVAs) in space or on land are presently expensive, risky, and 
highly dependent on large teams of ground operation support. The “Mobile Agents” (MA) system is 
designed to automate the role of CapCom in Apollo — including monitoring and managing EVA 
navigation, scheduling, equipment deployment, telemetry, health tracking, and data collection. Mobile 
Agents uses a multi-agent, distributed architecture , 1 ' 7 integrating diverse components in a voice-controlled 
EVA communications and astronaut assistant system , currently including: biosensors, cameras, GPS, 
EVA Robotic Assistant, and science database. This wireless computing and software system proactively 
transmits data, assisting communication between EVA crew, autonomous robots, vehicle or surface 
habitat crew, and remote support teams. In conventional information processing terms, MA provides a 
voice-commanded, workflow tool that proactively coordinates the actions of robotic devices and people, 
while facilitating strategic and scientific collaboration in human team s . 

Agents — incorporating models of EVA activities and robots/devices represented in a modular, 
reconfigurable, reusable way — store, forward, interpret, and transform data as they become available to 
help people and a robotic network cooperate to make operations more safe, affordable, and effective (e.g., 
agents could monitor EVA progress and consumables to dynamically replan EVAs). A spoken dialogue 
interface enables people to communicate with agents, including robotic controllers. A science database in 
the habitat is mirrored to Earth, for access and annotation by scientists and mission support for 
collaborative review and planning. 

The Mobile Agent Architecture (MAA) has been developed iteratively in the context of use, including 
six years of ethnographic observation 9 ' 11 and field experiments , 1213 with scientists doing authentic 
exploration. The design was inspired by analysis of Apollo lunar traverses , 14 ’ 15 as well as a two-week 
simulated Mars mission at the Mars Desert Research Station (MDRS ). 16 The MDRS field “testbed” 
facility/site provides a test bed for identifying requirements, competitively testing alternative 
technologies/protocols, and training astronauts . 17 

Using the Mobile Agents approach could reduce human monitoring during an EVA from dozens of 
full-time controllers to an alert-based system that contacts and involves people only during anomalous 
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situations (e.g., habitat loudspeaker broadcast and email to Earth). Target metrics relative to Apollo 
include: 100% reduction in manual telemetry readouts, 100% transmission of science data to Earth during 
EVAs, 80% reduction in human monitoring, and 60% automation of dynamic EVA replanning. The 
current system is ready for advanced application. However, replicating human CapCom capabilities in 
navigation interpretation and advising, EVA scheduling, and safety monitoring requires development of 
models with uncertain, judgmental inferences. Such capabilities can be gradually incorporated, but in the 
limit, expert-level human reasoning lies in the realm of open “artificial intelligence” research. By 
following the incremental cycle of analysis, design, implementation, and field experimentation, the 
Mobile Agents system provides EVA computing and communication tools that are always useful, with 
progressively improving capabilities. 

The MAA is applicable to many space work situations. For example, construction involves building 
and navigating maps (including localization and configuring equipment for terrain topology), interacting 
with piloted and unpiloted rovers, changing environmental conditions, and remote team collaboration. 

Subsequent subsections of this introduction provide an overview of objectives and accomplishments, 
a summary of the analog mission cycle, and the scenarios and schedule for the field experiments. 
Subsequent sections of this article describe Field Participants and the Mobile Agents 2004 Configuration, 
followed by three sections describing Crew Collaboration with the Support Team, Crew/RST 
Collaboration Tools, and Reporting during the Simulated Mission. More detailed sections are based on 
reports written in the field: Pedestrian EVA Scenario (Pooh ’s Corner), Head of Canyon Scenario 
Planning, and Long Relay EVA Scenario (Lith Canyon ). Finally, the article provides two summary 
sections: Results and Discussion (including design recommendations) and Conclusions. 

A. Overview of 2004 Objectives and Accomplishments 

Crew 29 returned to the Mars Desert Research Station (MDRS habitat, “the hab”), five miles from the 
minuscule town of Hanksville, tucked in the comer of the southeast Utah canyons and painted desert, for 
two weeks of experimental operation of the Mobile Agents system. In April 2003, we had shown as Crew 
16 how to use MDRS as its name suggests — a research station — a protected workplace for scientists and 
engineers to configure, deploy, and test sophisticated technologies and protocols in an extreme 
environment. 14 In 2004 we connected a database to the network in the hab (“ScienceOrganizer”) and 
made a mirrored web site accessible to a remote science team (RST), with a five-minute time delay to 
simulate interplanetary communication. The RST receives data from the field as it is collected during the 
EVA, plus they provide feedback on the crew’s plans for each day. 

Six people ate and slept in the hab (Clancey, Sierhuis, Alena, Dowding, Garry, and Semple) while the 
remaining group of 12 or so found accommodation in Hanksville. During the day, the habitat was shared 
by the full team, with the Ames and Johnson Space Center (JSC) computer scientists and SUNY-Buffalo 
geologists sharing the upper deck of the habitat. (The JSC team also operated out of a rental truck 
“office” parked outside the habitat.) Only the EVAs were conducted in simulation mode (with two people 
in suits, and a support crew off camera). We used collaboration tools to prepare EVAs, preparing 
diagrams, outlined notes, plans, and videos that could be shared with the RST; this was also performed in 
a kind of simulation mode, while the remainder of the formal crew of six prepared dinner and wrote 
reports (and others returned to Hanksville for the night). 

Our objectives this year were to demonstrate longer duration (> 30 minutes) and more distant (> 100 
meters) EVAs with the Mobile Agents system, including: 

o the EVA Robotic Assistant 18,19 (ERA, http://www.jsc.nasa.gov/er/era) integrated with astronaut 
operations 

o more sophisticated voice commanding for science data collection (images, voice annotations, and 
samples) 

o automatic association of science data with named (and automatically mapped) work sites 
o appropriate alerting for exceeding health, schedule, and location thresholds, including verbal 
warnings to the astronauts, as well as on the loudspeaker in the hab, and via email to the RST 
o collaboration with a well-organized remote science team 
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o planning and meeting replay tools for crew-RST communications 
o EVA data stored automatically in a database shared by the crew and RST. 

Our progress over the two weeks was recorded in daily reports 
( http://www. marssociety. org/MDRS/fs03 ) . We were host to several NASA managers, two video crews, and 
an international reporter team. During the second week the field experiments were presented on NASA 
TV’s Video File. 

The schedule was deliberately aggressive, with a week for each of two major sites. However, we had 
planned to take the hab into “closed simulation mode” after EVAs to document sample curating and 
mission planning and only did that once. The second week’s EVAs ended at such odd hours (sometimes 
after dark) and we were so tired, continuing our work in a formal way was impractical. Closed system 
simulation of planning, EVA, and review will have to wait for a more routinely usable system, certainly 
one not requiring a dozen support people and a mini-camp in the field. Nevertheless, Sierhuis always 
carried through the sequence of interactions with the RST (organized remotely by Rupert), and this was 
itself almost a second expedition on the side, with many participants, tools, and reports. We learned that 
having EVAs on sequential days was difficult to cooperatively plan and review, given the uncertainty of 
testing and multiple time zones. Detailed reports about the scenarios appear in the main sections of this 
paper. See Table 2 in the Results and Discussion section for an example “run” of the system. 

B. Annual Cycle of Analog Research and Development 

From a scientific perspective, an analog is any site (e.g., deep ocean hydrothermal vents) or object 
(such as a model in a wind tunnel) that can be studied first hand to learn about another place usually 
remote or impossible to study directly (e.g., Mars) or an artifact that is being designed (e.g., a space 
vehicle). In space exploration, analog sites were commonly used in Apollo for training astronauts (see 
Ref. 11 for a survey and the references in Ref. 9 and 10). Today, scientists, engineers, and flight 
controllers work together in analog sites for experimenting with tools and operations concepts. For 
example, analog sites were used to prepare for the Mars Exploration Rover mission during 2003. 

The use of analog missions is thus a research and development methodology’ that integrates operations 
and technology research. “Research” is used more broadly here than is familiar to some managers — just 
as the collaboration of flight operations practitioners and academic researchers may be unexpected. 
Research here refers to any scientifically studied phenomena in authentic work settings, to reveal the 
interactions of practice (e.g., how geologists prefer to explore), technology (e.g., a robotic system), and 
operations (e.g., protocols for communication between the crew and RST). Research of this kind, 
including observational and analytic studies, is an essential aspect of work systems design , which 
specializes in practically relating roles, procedures, facilities, schedules, and tools. 12 

Recently, computer scientists and flight operations researchers have chosen to follow geologists and 
biologists to Mars analog settings, where exploration can be studied and technology used 
experimentally. 20 22 After several years, we can identify an analog learning cycle, which iterates through 
at least five phases: 

1. Analyze Data (including previous field experience, historical analogs, and related 
work). 

2. Identify Operations Concepts (including new scenario and technology opportunities). 

3. Requirements and Design Workshop (>= 6 months prior to field work, including all 
project participants). 

4. Write Buildable Specifications (constrained by time, funding, personnel; including 
effort and time worksheets) 

5. Operational Readiness Test (at least one month prior to field test, bringing key 
personnel and equipment together for connectivity and communications tests in a 
conventional laboratory/research center setting). 
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6. Field Experiment in Authentic Work Setting (two weeks or more, with convenient 
living and work facilities, including observation and recording of simulated 
operations). 

Following this cycle since early 2002, the Mobile Agents project has progressed from connectivity trials 
to nearly practical application of the tool during actual geologic exploration. Figure 1 shows how the use 
of the tool progresses in phases from mere testing (which does not require authentic work settings) to 
exploration in settings that are relatively well-known to science, but new to the participants, to 
exploration" that involves authentic science (new, publishable discoveries). For most research and 
development purposes, analog experiments need only involve authentic exploration; however, some 
locations such as the Arctic and Antarctic offer the opportunity for new geology and biology research 
work as well. * 1 Note that Fig. 1 can be adapted for engineering work (e.g., systems monitoring). 
Engineering scenarios and configurations are being developed for Mobile Agents field experiments in 
2005. 



Figure 1. Relation of science, exploration, and analog experiments. 

Research in analog settings investigating a particular work system design shifts from 
1) controlled tests (in which the only authentic experience is using the tools in 
largely predefined settings) to 2) scripted exercises (involving authentic exploration) 
to 3) planned exploration goals and regions (involving authentic scientific 
exploration). “Science” here refers to geology, biology, astronomy, physics, etc. The 
scientific study of people and work systems for the design of space missions proceeds 
in parallel on another dimension. 

Furthermore, in the Mobile Agents project, we are investigating the use of simulation-based tools for 
designing field configurations and scenarios. 23 29 Such simulations could become part of the analog 
learning cycle, such that simulations are fine-tuned from experimental results, and potentially used for 
evaluating designs when analog experiments are not practical. A simulation using the Brahms tool can be 
transformed relatively directly into a runtime system, such that some computer agents are replaced by 
actual people and robots, and others become part of the software used by the astronauts, mission support, 
engineering, and science teams. 


“Exploration” in the context of space exploration involves getting to know a place, with typical phases of 
surveying, mapping, route finding, and region/boundary demarcation. 

1 Note that the word “science” is used here quite narrowly, which is common in the space exploration community, to 
connote the scientific study of life and the universe. In practice, the development of exploration tools involves new 
scientific research in many other disciplines, including anthropology, psychology, sociology, and of course 
computer science. More broadly, the design of flight operations systems involves a kind of scientific work that 
cannot be reduced to any of these disciplines (e.g., see the Results and Discussion section for a summary of 
requirements discovered empirically during Mobile Agents 2004 experiments). 
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An important consideration in Mobile Agents system development is the relation of the tools and 
scenarios to NASA’s exploration missions. As mentioned, it will take decades before we accomplish the 
vision of automating the role of Capcom. However, near-term applications can be tied to the requirements 
for the Crew Exploration Vehicle, emergency EVAs in space, and construction/maintenance in the next 
lunar expeditions. In this process, we relate the overall work to the MA design in three broad levels of 
analysis: 

1. Identification of Exploration Activities and Tasks (e.g., communications between remote 
science team and ISS crew). 

2. Determination of Fixed Constraints (e.g., Terrain, Timing, Crew Size, External Support) 

3. Specification of Design Parameters, for example: 

• Data and workflow systems 

• Tools: Drills, construction equipment, instruments 

• Suits, gloves, life support 

• Transportation 

• Operations protocols: Crew (vehicle/habitat and EVA), Support (engineers and scientists) 
Often, in developing tools for future missions, we proceed bottom-up, so we can establish a basic 
technology infrastructure. However, we expect that soon analog research and development will become 
more “middle-out” as we relate problems posed by actual mission requirements to the opportunities 
afforded by existing analog sites, facilities, and technological capabilities (see the Conclusions section). 

C. Scenarios and Schedule Overview 

The 2004 field experiments were organized for two distinct settings, one week each. Each setting 
required completely different deployment of the wireless network backbone, with 1-2 days required for 
reconfiguration. Each setting involved multiple days of work, with different scenarios involving very 
different interactions between the two geology “astronauts”* and the EVA robotic assistant. The scenarios 
are reviewed briefly here; more details appear in subsequent sections. 

The first week involved a pedestrian EVA to “Pooh’s Corner,” a site of interest to the geologists, 
identified by a brief visit in civilian clothes (called a “baseline study”) in 2003 (see Fig. 2). On the first 
day, the ERA performed an “autonomous” seven hour reconnaissance, following a route with 
predetermined waypoints and transmitting video, photographs, and panoramas along the way. An EVA 
plan was developed using data returned by the ERA and available aerial maps. The plan was represented 
using computer tools, shared with and annotated by the remote science team (RST), and then uploaded 
into the Mobile Agents system as a sequence of located, timed activities. On the next day, the astronauts 
explored on foot and sampled locations according to the activities represented in the plan. The plan 
provides expectations to the Mobile Agents system, such that it monitor the schedule and the astronauts’ 
location, providing appropriate alerting to the crew on the EVA and in the hab. 

The second week experiments occurred at “Lith Canyon,” a site 5 km away, also involving multiple 
scenarios. On one day, the ERA was left at the head of the canyon, moving between predetermined 
waypoints and transmitting photographs and video that enabled the habitat crew to monitor the astronauts 
in the canyon. On a much longer five hour EVA, the ERA was lowered by block and tackle into the 
canyon, so it could accompany the astronauts — serving as a network relay, carrying samples and tools on 
a cart, plus taking photographs and panoramas on voice command, as well as video-tracking an astronaut. 
The ERA successfully followed the astronauts approximately 150 m, at a rate of 0.15 m/s with laser-based 
obstacle avoidance (0.5 m/s without obstacle avoidance on relatively flat terrain). 

The voice commands introduced for 2004 included: 

• Boudreaux 8 , follow/watch/take a picture of me 

• Boudreaux, take a panorama 


* We will refer to the geology graduate students as “astronauts” so the roles being played are clear. 

8 The ERA team has developed multiple robots. Experiments in 2004 used the ERA named “Boudreaux.” 
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• Boudreaux, execute your plan/come home 

• Label the last image/collection as X** 

• Start activity walk/work/survey/sample {at location Y/here} {for time T} 

• Change the duration of the current activity to X seconds/minutes/hours 

• Associate voice note/image/sample bag {called X} with {the last} sample bag/image/voice 
note {called Y} 

• Create sample bag LL/DD/DD" 



Figure 2. MDRS Topographic Setting and Scenario Relay Configuration. R1 designates the 
antenna on Pooh ’s Corner hill, providing a wireless LAN relay in first week’s scenarios. R2 and R3 
designate antenna placement on ridges (“WLAN Backbone”) for Lith Canyon scenarios in the second 
week, approximately 5 km from MDRS hab. An ATV served as an access point on the opposite side of 
the canyon, with the ERA following the astronauts providing the final relay on the canyon floor. A 
TES trailer-based satellite system, connected by Ethernet cable to MDRS, provided internet access to 
NASA Ames, where the science data were mirrored for access by the remote science team. 


• Call my location work site DD 


Names in the 2004 configuration must be one of a set of pre-determined vegetables (e.g., “asparagus”) or by 
saying “work site” followed by a number. 

" L is a letter specified by a predetermined vocabulary, e.g., “Whiskey-Lima” = WL, which designates “west Lith”; 
D is a digit (0 - 9). 


II. Participants in Mobile Agents 2004 Field Experiments 

Over fifty people participated in this project during the two week 2004 field experiments. Twenty-two 
people worked on-site at the Mars Desert Research Station, including the crew who lived in the hab and 
provided regular outreach reports (Clancey, Sierhuis, Alena, Dowding, Garry, Semple). These people 
came from three NASA centers and six universities: 

• NASA Ames Research Center (Moffett Field, CA) 

o Brahms Modeling and Simulation System ( www. agentisolutions.com) : Clancey and Sierhuis, 
Co-PIs; Ron van Hoof, Michael Scott, Yilmaz Cengeloglu, Charis Kaskiris (UCB), Jeff 
Bradshaw (Institute of Human and Machine Cognition, University of West Florida, 
Pensacola, FL) 

o Mobile Exploration System (MEX, wireless computing and communications system): 
Rick Alena, PI; Charles Lee, John Ossenfort, Ed Walker 
o RIACS Language Interfaces and Speech Technolog y group (RIALIST; 

www.riacs.edu/speech/speech.html ): John Dowding 
o NASA Research & Education Network (NREN; www.nren.nasa.gov/ ): Ray Gilstrap, Thom 
Stone 

• NASA Glenn Research Center (Cleveland, OH) 

o Michael Cauley, David Pleva, Mark Seibert 

• NASA Johnson Space Center (Houston, TX) 

o EVA Robotic Assistant (ERA, aka “Boudreaux” and “the robot”): Jeff Graham, PI; 
Kimberly Tyree, Nathan Howard, Robert Hirsh 

• SUNY Buffalo (State University New York at Buffalo) 

o Abigail Semple, Brent Garry (Geology Department) 

A group of at least 16 more people were part of the Remote Science Team, led by Rupert and Sierhuis: 

• Remote Science Team — Geologists and Biologists 

o Shannon Rupert (MiraCosta College, Oceanside, CA) 
o Stacy Sklar (Northern Arizona University) 

o Melisa Farley, Kyle Frederick, Brett Burkett, Shannon Kobs, Nadine Calis, Amy Webb 
(SUNY Buffalo) 

• Remote Science Team — Collaborative Tools Support 

o Compendium : Simon Buckingham Shum, Michelle Bachler, A1 Selvin (Knowledge Media 
Institute, Open University, Milton Keynes, UK) 
o BuddySpace : Marc Eisenstadt and Jiri Komzak (Open University) 

o Meeting Replay tool : Danius Michaelides, Kevin Page, Dave De Roure and Nigel 
Shadbolt (Intelligence, Agents, Multimedia Group; Univ. of Southampton, UK) 
o ScienceOrganizer: Dan Berrios, Ian Sturken, David Hall, Rich Keller (Code IC, NASA- 
Ames Research Center) 

Other support behind the scenes was provided by NASA Ames Public Affairs, the RIACS 
administration at Ames, and the Northern California Mars Society volunteers who provided logistic 
“mission support” (plus Tony Muscatello in Colorado, representing the Mars Society). Frank Schubert 
and Don Foutz (owner-manager of the Hanksville Whispering Sands Motel) maintained and resupplied 
the utilities (power, water, septic, ATVs). 

III. Mobile Agent 2004 Configuration 

Here we describe the configuration of software and devices, called the Mobile Agents Architecture 
(MAA), used for the 2004 field experiments. The MAA changes from one year to the next, with different 
configurations used for scenarios within a given field season. For example, during the Pooh’s Comer 
reconnaissance in 2004, the astronauts were inside the hab and not giving voice commands to the robot; 
instead they monitored its location and the returned data on computer screens. Each MAA configuration 
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consists of revised capabilities for the existing software agents, new software agents, new software 
systems (e.g., crew planning tools), new voice commanding vocabulary and processing, and new devices 
(e.g., a panorama camera). 

A. Background: Software agents and Multiagent Systems 

The term software agent is used in different ways in computer science. Software agents, also called 
“intelligent agents,” were invented in the 1970s by artificial intelligence researchers as programs that 
could assist computer users, such as for finding resources on the internet. 30 A software agent is essentially 
a computer program with goals, models, inferential capability, and a memory of “beliefs” (propositions 
about the world), designed so people can interact with it as if it were an independent entity. Starting in the 
1980s, this approach became particularly useful when software agents were able to communicate over the 
internet, serving as “personal agents” 31 that manage the interests of different people. 

In the 1990s, the software agent representational approach was strengthened by the advent of multi- 
agent systems, essentially systems of coordinated programs within some network and often geographic 
setting. 32 With the invention of internet browsers and their nearly universal use for commerce and 
research, plus the availability of much faster computers with more memory, the multi-agent approach 
became adopted for a variety of practical applications. In the Mobile Agents project, we go a step further 
by implementing the agents on portable computers on a wireless network. In considering the benefit of 
software agents in the design and implementation of the Mobile Architecture, especially as a means of 
integrating a diversity of systems with people in a work system, we believe that the software agents 
approach is not simply a way of programming, but a new paradigm for software development. 

B. MAA Technical Overview 

The Mobile Agents Architecture consists of four software foundations: the Java programming 
language, the Brahms multiagent modeling and programming language, 23 ’ 24 the KAoS agent- 
communication framework, 33 and CORBA. Together these software tools allow us to implement the 
MAA distributed over a wide variety of wirelessly networked computers, from the astronaut spacesuit 
computers, to the robot computers and the computers in the MDRS habitat, to the computers for the 
Remote Science Team (RST) back on Earth (Fig. 3). 

In all that follows, the reader should remember that no particular MA configuration represents a 
recommendation of “what should be,” but rather is always a compromise for what funding allows, what 
time and personnel available permit to be built and tested, and the technology available and convenient to 
deploy. What is important, for example, is not the use of USB for connecting a camera to a computer or 
the use of GPS, but the concept and operation of the multi-agent system as a means of managing 
exploration, scientific, and engineering work in logistically challenging environments. The essential 
elements of the MAA are: multi-layer communication network 34 (i.e., LAN, agent registry, distributed 
agents); distributed computers communicating with each other and attached peripherals; use of APIs to 
integrate peripherals with the agent system; personal agents managing workflow (including agent 
monitoring, advising, and data management); crew commanding and feedback protocol; 35 automated 
association of time, location, and activity data with science data; 36 activity-based planning (vs. task-based 
approach 26 ); tools to facilitate remote collaboration between crew, support, and RST team s . 
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Figure 3. Mobile Agents 2004 Network and Components Configuration. Notice that the ERA serves as relay 
to the astronauts, which enables extending EV As around corners and into craters or ravines. 


Figure 3 shows the hardware, network and software elements in MAA 2004. Starting at the right side 
of the graphic we see the two EVA astronauts with their backpack computers. Each backpack computer 
runs the Brahms software agents that integrate and communicate with all the other software pieces in the 
architecture. The agents coordinate the dataflow between the systems as well as the work process of the 
astronauts. The dialogue system software allows the astronauts to speak to their personal software agents 
using speech in a predetennined vocabulary with flexible phrasing. The GPS system gathers location data 
from differential GPS (dGPS) units with 2 cm accuracy. The biovest system gathers astronaut health 
information from health-monitoring sensors. Digital cameras can be connected to the backpack computers 
so software agents can download digital image files taken by the astronaut and associate them with other 
gathered data (such as the astronaut's GPS data), as well as store and forward these images to the habitat 
and Earth. 

Next, one of the ERA computers runs the Brahms software agents supporting the ERA in executing 
plans and allowing the robot to communicate its information to the Brahms software agents on the 
astronaut’s spacesuit and in the habitat. The ERA software agents allow us to easily integrate the ERA’s 
capabilities with the rest of the software in the system. The ERA can be used in two different modes, an 
autonomous mode and an interactive team mode. In the autonomous mode an ERA software agent 
executes a previously uploaded EVA plan. This allows the robot to go on EVAs by itself and take 
panorama and other images that are immediately communicated back to the habitat. In the interactive 
team mode the ERA supports the astronauts on their EVA and gets its commands either via a speech 
command from the astronauts or a crew member in the habitat (called the habitat communicator, 
HabCom), or from a graphical user interface (GUI) operated by HabCom. The ERA can take digital 
panorama images, find its way to named GPS locations, carry the astronaut’s tools and science samples, 
print a curation label for the rock and soil samples gathered by the astronauts and autonomously follow an 
astronaut. The ERA robot can also function as a network relay node for the astronauts, allowing the 
astronauts to wander to interesting areas providing network connectivity back to the habitat. 
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An all-terrain vehicle (ATV) serves as a wireless network “hub” between the astronauts, the ERA 
robot and the habitat (Fig. 3). Using this approach we can reconfigure the communication path for the 
dataflow between astronauts, robot and habitat. The ATV computer serves as the MAA agent directory 
service, allowing the agents to find each other on this distributed network. The KAoS software that is part 
of the Brahm s agent software environment in effect creates a distributed agent framework allowing the 
MAA software agents to be distributed and communicate over a 5 km wireless WAN. 

Inside the MDRS habitat a set of software agents serve as the central coordination, communication and 
storage facility integrated with a number of other software systems previously developed to store science 
data, and allow distributed teams to collaborate asynchronously (this is important because the 
communication time-delay between Earth and Mars is too long to permit synchronous collaboration). This 
integrated habitat software environment allows the crew to collaboratively, but asynchronously, create 
EVA plans with the RST and analyze science data in the habitat after an EVA. Software agents running 
inside the habitat automatically monitor and store science and EVA plan data that is communicated by the 
software agents of the astronauts and ERA. After storing this data, a software agent automatically emails 
EVA data to the RST back on Earth via a satellite connection to the internet. At the same time the data 
stored inside the habitat's ScienceOrganizer system 36 is automatically copied to a ScienceOrganizer 
database back on Earth. This allows the RST to access the data in near real-time (with an artificial five 
minute delay). EVA plans are created and communicated by the crew to the RST using the Compendium 
software tool (http://www.compendiuminstitute.org) that is specifically developed for discussions and 
modeling problems in the form of hypermedia concept maps. This Compendium tool is used to allow the 
crew and the RST to collaborate. A Compendium software agent in the habitat can read the EVA plan 
from the Compendium database and distribute the plan amongst all the software agents in the MAA. This 
agent also stores all science data that comes into the habitat in the Compendium database, so that the crew 
and the RST can discuss the data within the Compendium tool. 

Figure 4 gives a graphical view of the software agents within the MAA. The solid icon circles in the 
three black ovals show the software agents running inside the Brahm s virtual machine on the astronaut’s 
backpack, the ERA robot and on the HabCom computer inside the hab. 

Figure 5 presents an example of the information flow using proxy agents (a form of interagent 
communication used in the 2002-2004 field experiments). Astronaut one says that he is starting the 
activity of sampling an area. This is processed by Astronaut one’s dialog agent and passed as a 
communication to his personal agent (PA). From here, the information needs to be communicated to the 
HabCom PA, which occurs via the HabCom PA proxy that co-exists in the Brahm s system running on 
Astronaut one’s backpack. The proxy uses information provided by a central registry (in this Mobile 
Agents configuration) to find the HabCom PA running on a platform in the hab. From the HabCom PA, 
the information is communicated by email to the RST and by a loudspeaker to the HabCom (a crew 
member in the hab) via the HabCom dialog agent (which converts the text to speech). HabCom PA then 
provides feedback to Astronaut one and two that the activity has begun, first using proxies to find the 
corresponding PAs, from which the information is communicated to the corresponding dialog agents, and 
then spoken to the astronauts on their headphones. All of the transmission of requests and scientific data 
follows similar paths between the crew members and their PAs. 

Note that biosensor and GPS data is handled more directly for efficiency by pulling data as needed 
from external sources, rather than using the agent communication network (which is to say that agent 
communications do not replace telemetry networks). Furthermore, there is nothing fixed about the design 
of proxies and PAs shown here; this is just one of many possible ways to partition computational 
procedures and information flow. See the Results and Discussion section for related comments about 
improved designs. 
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Figure 4. Brahms software agents in the Mobile Agents 2004 Configuration. 
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IV. Crew Collaboration with Remote Science Team 

The small Mars crew will undoubtedly collaborate with groups of scientists back on Earth. How this 
collaboration will happen is a matter of conjecture and experimentation. The Mobile Agents framework 
provides a means for implementing a computer-supported Mars- and Earth-based science work system, 
which we first employed in 2004. This system includes both the human work practices and computer 
tools with dataflow management systems. Here human-centered design meets work process design. In the 
empirical design approach we are using in the Mobile Agents project we are guided by the capabilities of 
the people and their objectives. People are at the center of the total system, and people are supported in 
their work by computer tools. This can be contrasted with a top-down functional approach, which 
attempts to “optimize” the work to be done, automating as much as possible, developing corresponding 
tools, and then training people to deal with the resulting work processes and interfaces. Instead, we 
develop automation that supports the human work practices, allowing people to do what they do well, and 
to work in ways they are most familiar. 

We start simply, asking basic questions such as how the Mars crew can communicate their daily EVA 
plans and captured science data during and after an EVA back to their colleagues on Earth. This leads us 
to question what the role of an Earth-based science team should be. Can the RST participate in the 
planning of daily EVAs on Mars? Will the RST be able to get the science data in time to make useful 
suggestions to the crew? Will the RST be able to follow the field crew’s investigations? Will the crew be 
able to absorb the RST suggestions in a timely manner to develop a daily EVA plan? How will the RST 
EVA plan compare with the crew plan? 

For the initial experiment in 2004, we defined a relatively simple science work process, using three 
pre-existing domain-general software tools. We integrated two of the three tools with the MAA, enabling 
the automatic flow of data between Mars and Earth. With this integration we showed that it is easy to add 
new software tools to the MAA. Previously developed software tools and systems can interact with 
software systems already in the architecture without having to develop dedicated software interfaces 
between them. This important feature of the MAA allows us to freely include separate tools to support 
specific functions in the work process. 

We briefly describe the make-up of the RST and the crew/RST workflow process. We then describe 
how the three tools support the crew and the RST in doing their work. 

A. The Remote Science Team 

The RST for the 2004 field experiments was highly distributed, consisting of scientists from all over 
the world. Figure 6 shows the RST organization. The team had four parts, one located in San Diego, New 
York, Arizona, and England. Rupert coordinated these parts, plus was point of contact between the Crew 
Uplink Lead (Sierhuis) and the RST. The SUNY Buffalo team consisted of faculty and geology graduate 
students from the Volcanoes Studies Group. Sklar, a student at Northern Arizona University, led a team 
made up of geologists from the MDRS RST, volunteer scientists who supported all of the 2004 MDRS 
rotations and have participated as field crew. Buckingham Shum and Selvin of The Open University in 
the UK led the meeting facilitators, who managed communication between the crew and the RST through 
Compendium and Meeting Replay. 

The RST teams collaborated to analyze the EVA science data and provide feedback on the crew’s next 
EVA plan. We defined a simple work process in which the crew planned and executed their EVAs in a 
daily EVA-cycle using several tools. Because the RST members were so distributed, their interaction also 
benefited from collaboration tools. The RST Facilitator facilitated web-based teleconferences between the 
RST teams. The workflow process is described next. 
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Figure 6. The Remote Science Team. Mars Crew inset corresponds to Fig. 4. 


B. The Crew/RST Workflow Process 

The workflow process is divided into two parallel processes that are dependent on each other. Figure 7 
shows a high-level workflow process model. 



Figure 7. The Mobile Agents Project Crew/RST Workflow. 

The idealized flow shown in Fig. 7 is as follows: After the EVA, the crew analyzes the collected 
science data (rock and soil samples) back in the hab. After this analysis, the crew discusses the results of 
the day’s EVA and plan the EVA for the next day. This meeting is facilitated with the Compendium 
meeting capture software and is video taped. The result is communicated to Earth. On Earth, the crew’s 
meeting video and Compendium database is used in the Meeting Replay tool. This allows the RST to 
view the EVA planning discussion of the crew and at the same time view the crew’s plan in the 
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Compendium tool. After the individual RST members have viewed the video, they have a remotely 
facilitated meeting on the web, called the Science Operations Work Group meeting. The name is taken 
from a similar meeting held during each sol on the Mars Exploration Rover (MER) mission at JPL 
(http://origin.mars5.jpl.nasa.gov/home/). During the SOWG meeting on the MER mission, science 
subgroup leaders review and assemble the plan for the next day, relating timing and resources. Similarly, 
the RST team meets together in a teleconference to review the plan made earlier by the crew on Mars. 
This meeting is also captured in the Compendium meeting capture tool. The result is emailed back to 
Mars before the crew wakes up the next morning. The next morning at 8:30 (local Mars time) the crew 
reviews the RST plan in Compendium, and decides on the final plan for the day’s EVA. 

More details about the collaboration tools are provided in the following section. 

V. Crew/RST Collaboration Support Tools 

Mobile Agents 2004 introduced a new strand of research in MDRS analog research, concerning how 
to support the Remote Science Team (RST), which in actual missions is likely to be spread across the 
Earth in multiple time zones. Just as the MER 2004 mission showed after a few months, it is impractical 
to physically co-locate all relevant experts who might be needed in a multi-year Mars mission. Looking 
ahead a decade or two, this research prototypes and evaluates tools to support scientific teamwork under 
such circumstances, advancing the state of the art in information networks. 

To this end, a collaboration was formed between NASA-Ames’ work systems design research group 
and the UK’s CoAKTinG Project. 37 CoAKTinG (“Collaborative Advanced Knowledge Technologies in 
the Grid”) is part of the Advanced Knowledge Technologies consortium and is developing new kinds of 
collaboration technology to support what they call e-Science. MA 2004 provided a testbed for some of the 
CoAKTinG tools, to see how well they could connect the RST with each other, with a crew living and 
working out of conversational contact. 

A. Preparation for Field Experiments 

To prepare, the RST held weekly planning meetings via teleconference for over five months, 
exploring possible science questions that could be addressed by the field crew during the field test and 
putting the new collaboration support tools through their paces. Thus we began to understand the role that 
different collaborative technologies could play in the overall work ecology of the RST and astronauts. 

In particular, during the preparation the RST used the ScienceOrganizer system to archive and to li nk 
meaningfully their project materials and experimented with Compendium to design their own maps to 
support a geological methodology. A trial version of the Meeting Replay tool from the Operational 
Readiness Test in March 2004 also generated much interest, and was fine-tuned as a result. The 
BuddySpace messaging and awareness tool was released once the MDRS work had started. These RST 
Communication System tools are described in subsequent sections. 

B. ScienceOrganizer: Long-term, formal memory for mission data and knowledge 

ScienceOrganizer 36 was developed at NASA Ames, and provides a web-based semantic database that 
can be populated and searched by scientists or engineers. It has been previously used to support the 
Columbia Shuttle accident investigation, 38 to study patterns of malaria in Africa, and to design and 
execute experiments on microbes, among other applications. ScienceOrganizer provides a rich interface 
for semantically linking resources (i.e., a relational database with well-defined descriptive relations, such 
as “plan activity”), and is most suited for long-term knowledge management, especially for distributed 
teams collaborating asynchronously. Figures 8 and 9 illustrate how a voice note is stored in 
ScienceOrganizer and the notification is received by a member of the RST. 
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From: capcom@agentisolutions.com 

Subject: MDRS New VoiceAnnotation: voice_note_9 
Date: May 7, 2004 3:47:36 PM PDT 
To: mdrs rst@agentisolutions.com 

New VoiceAnnotation: voice_note_9 

EVA Plan: LithCanyonSegmentThreeEVAPlan 

Activity: WorkAtWayPointl6 

Creator: AstroTwo 

TimeStamp: 05/07/2004 23:47:12 

File Name: voice_note_2004-4-7_23-46-28.wav 

Latitude: 3827.3077 NORTH 
Longitude: 1 1047.4074 WEST 

Northing: 4256335.2693082085 Easting: 518312.1563357575 Zone: 12S 

ScienceOrganizer Link : https://marst.arc.nasa, gov/org/1660 1 5 

Figure 8. Email sent to RST indicating a new voice note. Besides the time and precise GPS location, the email 
indicates that astronaut 2 created the voice note while working at waypoint 16 during the Segment 3 EVA in Lith 
Canyon. This information is associated with the voice note in ScienceOrganizer, at the specified URL. 


ScienceOrganizer: An Information-Sharing Tool for Mobile Agents 
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Figure 9. A ScienceOrganizer database entry showing a voice annotation created Lith Canyon EVA. 

On the left are hyperlinks to related resources, such as the location the voice note was taken, the astronaut, and the 
EVA itself. This is the page referenced in the email sent to the RST shown in Figure 8. 


Because the ScienceOrganizer interface consists of web pages, the system can be made available to 
virtually anyone, anywhere. However, it was not designed to support meeting capture, ffee-form thinking, 
and brainstorming. One of the questions we investigated was how ScienceOrganizer could be 
complemented by other collaboration tools such as Compendium. 


17 


C. Compendium: Mapping meetings as group memory 

Meetings are as pervasive an activity in scientific life as elsewhere, and similarly, can often also be 
ineffective. Moreover, when conducted in an international context via a virtual environment such as a 
video- or tele-conference, many of the normal non-verbal cues that add fluidity to interaction are missing. 
The Compendium tool being developed at the Open University provides a way to focus discussions on 
specific topics, integrating other documents into the idea network that grows as contributions to a 
discussion are mapped on screen. Compendium differs from other concept mapping tools in the specific 
visual language it provides to capture and organize issues in real time as they emerge in discussions, and 
in the hypertext and semantic tagging mechanisms it provides to manage large databases. 

Compendium played several roles in the MA 2004 mission: 

1. Mediate between the hab crew and software agents. The Crew used Compendium to plan the 
route of the next EVA (Fig. 10), constructing a visual map of the locations they wanted to visit and the 
activities to be conducted at each. These Compendium maps were then interpreted by the agents that 
coordinated commands and the flow of information during an EVA. 
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Figure 10. An EVA plan constructed by the Crew in Compendium. 

This plan was read by the Brahms software agents that coordinated the EVA work flow. 


2. Compendium was used to view all the data gathered on the EVA when the Crew returned to the 
hab. All the data generated during an EVA (videos, photos, voice annotations...) streamed back over the 
wireless network and were stored in the hab in ScienceOrganizer and Compendium. Compendium 
displays data in concept maps (Fig. 11), assisting analysts in seeing and navigating certain kinds of 
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structure, and linking in their own ideas and documents. Compendium maps were sometimes shared by 
storing then in the ScienceOrganizer database, and data referenced by maps were archived in 
ScienceOrganizer. 


□ [Map]: EraModeLPANORAMA_IMAGE_1 


<t> 


eva 


T E 

1 2 1 — 1 3 

Boudreaux EVA To Poohs Corner 



Figure 11. Following the EVA, a map is created and populated in Compendium rendering the data 
gathered. Shown is an example Science Data map, highlighting the data gathered by Astronaut One. 


3. Compendium was used by the crew and RST to discuss the implications of the data. 
Compendium displays information in visual concept maps to support sense-making activities such as 
planning, discussions, and the gradual structuring of ideas. As the discussion unfolds, the contributions 
are simultaneously mapped on the screen (projected in the hab in a crew meeting, or shared over the 
internet in an RST teleconference). The RST could therefore see what options the crew may have 
discussed but rejected, and why, in order to provide scientific feedback to the crew (Fig. 12). 

The insight that the RST gains into the crew’s decision-making is further enriched by an indexed, 
digital video record of the crew’s meeting which integrates Compendium with other resources, as 
described next. 
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Key feedback from RSI? 



With respect to the pale grey.vrtiite layers 
you found, follow the Stratigraphy 
methodology f i urn the Outcrop 
perspective - detail is at your discretion, 
with rationale foe this delivered to RST 



We believe the diff. between the 3 ground 
colourings in Pan.1 are just soil types 
(hut need to check this on video to he 
sure) 




We have added more detailed 
annnotations to your EVA Analysis map, 
from which these summary points are 
taken 


Po 


T ®, 


Suggestions from RST? 


Where to go on Pan.2? Suggestions.. 


Gully between Rock Hill and Little Red Hill: 
determine if ancient fluvial envrnt ■ look 
. for signs of water 


t (D 2 

Stratigraphy on another lull (name to be 
identified): use the stratigraphy 
methodology 


Figure 12. Compendium map summarizing the RST’s feedback to the crew. Each node is hyperlinked to the 
map (i.e. the context) in which it was created. 


D. Meeting Replay: Enabling the RST to review a crew meeting 

Given the communication delay between Mars and Earth, the usual electronic ways of working 
together at a distance, such as conversation and the sharing of computer screens, are impractical. As part 
of the CoAKTinG project, the University of Southampton have developed a meeting replay tool, which 
combines meeting materials within an interface structured to enable quick and easy indexing navigation 
of the meeting record. 

During the mission we recorded the crew’s daily EVA planning meetings and delivering a replay of 
the meeting to the RST within a few hours. By experimenting with these techniques we hoped to see if 
the RST could gain a better understanding not only what a crew is deciding, but why, and how, in order to 
provide the best kind of feedback. 

Figure 1 3 shows the web-based Meeting Replay tool. The upper region shows the video of the meeting 
and the Compendium map as the discussion progresses. The lower region contains summary information 
about the meeting — who was there, who was speaking, the agenda, and an overview of the current topic 
(derived from the Compendium map). Some of this information is presented as a timeline, providing a 
visual index for an RST member to navigate the video, jumping to relevant or interesting parts of the 
discussion by clicking on the timeline or moving the slider. 

Before the RST meets collectively, each participant runs their own meeting replay. Should the need 
arise to review a specific point in the meeting as a group, one member can take control of all the meeting 
replays using the GroupSync feature. As that member navigates around the meeting everybody else’s 
client follows their lead, remaining in synchronization as long as required. 
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[Mop]: Poohs Corner DoyOneEva Crew Planning Meeting 04/27/04 
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Figure 13. Web-based meeting replay tool screenshot. When reviewing the meeting replay, Compendium has been 
extended so that it can be used as a ‘visual contents page ’ into the video. For instance, if the RST wants to see 
discussion prior to the recording of a particular decision, one can now click on this node in Compendium and the 
replay jumps to the point in the meeting where that node was recorded, and starts to play the video. 



E. BuddySpace: Instant messaging meets presence visualization 

Teams distributed in time and space are less aware of their colleagues’ activities and availability, 
giving a reduced sense of presence compared to being co-located. One challenge is how to characterize 
presence, as something that can be managed and visualized easily, while remaining consistent with the 
user’s expectations and work habits, including existing patterns for using Instant Messaging (IM) and 
other communication tools. 

A prototype presence management tool called BuddySpace has been developed at the Open 
University, which overlays presence information onto visualizations, both geographical (e.g. a map of a 
building, region or planet), and conceptual (e.g. a workflow chart, project plan, or experiment). 
(BuddySpace is based on Jabber/XMPP, and is interoperable with MSN Messenger, Yahoo, ICQ, and 
AIM.) In the RST BuddySpace map (Fig. 14), the availability of members is shown mapped to their 
geographical location. 

The preparation and simulated EVA experience suggested that ScienceOrganizer, Compendium, 
Meeting Replay, and BuddySpace used together is a powerful suite of tools for supporting RST-Crew 
collaboration. Reporting at a mission web site provided important means of communication between the 
crew, RST, mission support, and the public. These reports are described in the next section. 
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Figure 14. Prototype RST BuddySpace map, in which the online availability of colleagues is superimposed on 
locations in the US and UK. Clicking on one of the presence dots opens a conventional instant messaging session. 


VI. Reporting During the Simulated Mission 

The Mars Society maintains an archived set of crew reports throughout the multiple year campaigns at 
the Flashline Mars Arctic Research Station (FMARS) and MDRS. Each crew page includes biographies, 
mission information, and daily reports with photographs. A good part of this article was prepared from 
reports written and posted during the two weeks field experiments (see 
http://www.marssociety.org/MDRS/fs03). 

Each member of the official MDRS29 crew wrote reports according to his/her role: Commander 
“check in” reports provided daily synopses about the status of system deployment and testing, as well as a 
window on living and working at MDRS during this period. More complete EVA reports were written by 
the geology-astronauts. An EVA Communications Systems Report described the interaction of the crew 
with the RST and other aspects of the field work. Other crew reports included: Engineering (e.g., 
generator status), Greenhab (a gray-water recycling system), plus Flealth and Safety. To complement 
these reports, the RST Communication System Report was written by the RST Facilitator (Shum) in the 
UK, describing the experience and status of the collaboration tools. The RST report written by the RST 
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Lead (Rupert) detailed the experience of the RST team in understanding and commenting on EVA plans 
and data. 

Some of these reports are excerpted and adapted in subsequent sections, which describe the primary 
scenarios of MA 2004 and what occurred. 

VII. Pedestrian EVA Scenario (Pooh’s Corner) 

The first scenario involved several sequential activities: Reconnaissance by the ERA, crew-RST 
planning, and a pedestrian (walk out of the hab) EVA. These activities were planned for two contiguous 
days, though weather required a break in between. 

A. Day One of the Pooh's Corner EVA: ERA Reconnaissance and Crew Planning 

The first day plan was to send Boudreaux (the ERA used in 2004) over the previously planned path to 
Pooh's Corner to take some panoramas. After we looked at the available aerial image of the area around 
the hab that includes Pooh's Comer, we decided that we would create a plan that would automatically 
drive Boudreaux to Pooh's Comer and take four panorama images of the area. Figure 15 shows the 
discussion mapped in the Compendium tool on April 26, including a sketch of the plan for the ERA’s 
operation the next day. 



Figure 15. Morning briefing notes and plan in Compendium, for first day Pooh's Corner ERA 
reconnaissance. 

The actual EVA plan was also created in our Compendium meeting capture tool (Fig. 16) from 
waypoints that had been previously determined by walking through the site and logging GPS locations 
(work planned for 2005 will seek to eliminate this step, which of course contradicts the overall notion of 
“autonomous” robotic reconnaissance). After the waypoint plan was created in Compendium, it was 
automatically translated into a MA plan that could be interpreted by the ERA’s personal agent. 
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Figure 16. Plan created for ERA’S reconnaissance at Pooh’s Corner. Each waypoint icon designates an object 


containing additional information about location and actions. This representation can be automatically interpreted 
by a program that generates a plan represented in Brahms that is used by mobile agents during the EVA. 



Figure 17. Kim Tyree monitoring Boudreaux’s traversal inside 
MDRS with video, mapping, and other telemetry from the ERA. 


The Mobile Agents system was 
started, the plan was read in by the 
HabCom agent and distributed to 
the ERA Personal Agent. 
Boudreaux required about 5 hours 
to execute the plan autonomously. It 
eventually arrived at Pooh’s Comer, 
while the crew monitored its 
progress via Boudreaux’s video 
camera (Fig. 17). Excitement was 
highest when Boudreaux took its 
first panorama at waypoint 10 at the 
entrance of Pooh’s Comer. It took 
about 12 minutes to take the 
panorama and stitch it together into 
one image (Fig. 18). The picture 
was stored in ScienceOrganizer 
(Fig. 19) in the hab and then copied 
to the mirror site at Ames. Email 
was automatically sent to the RST. 
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Figure 18. Boudreaux's first panorama at entrance of Pooh's Corner (April 27, 2004) 
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Figure 19. Page in ScienceOrganizer corresponding to panorama (Fig. 18). 


Boudreaux returned to MDRS after driving more than 1 km. The reconnaissance was successful, 
although we did not receive all four panoramas as planned because of a communication error on the third 
attempt, which propagated to prevent the fourth instruction from working. As in operating the MER rover 
on Mars, such problems can usually be resolved and the software updated during the mission. 

While Boudreaux was returning, the crew analyzed the panoramas and discussed a plan for the next 
day’s EVA to the same site. Figure 20 shows the Compendium map of that preliminary planning 
discussion. This information was sent to the RST using the Meeting Replay tool, with the expectation that 
the RST would annotate the plan. 


B. Day Two EVA to Pooh's Corner: Astronaut EVA 

Before the 2004 field season, we had made a detailed schedule of the two weeks. After the ERA’s 
reconnaissance of Pooh’s Comer, the astronauts were planned to return to the site with the robot the next 
day to perform geology exploration. However, severe winds forced the EVA to be rescheduled a day, 
which provided an additional day of lab-bench testing and refinement of the EVA plan. 
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Figure 20. Crew’s analysis of Pooh’s Corner reconnaissance (excerpt) with next day EVA planning. 


The weather April 29 was cold and cloudy, permitting the EVA to proceed. But when the ERA team 
turned on Boudreaux its motors were not working. An hour delay (until noon) was anticipated, however at 
12:30 PM we decided that the astronauts should prepare, and we would do the EVA without the ERA if 
necessary. Then it began to rain. At 1 PM the rain had stopped and we started to suit up the astronauts. 
Preparing the new backpack with dGPS and two laptop computers, as well as getting two microphones 
fitted in the astronaut’s helmet (one for the speech dialogue system and one for the crew voice loop) and 
starting the Mobile Agents system required an hour. Coincidentally, HabCom reported that the plan was 
now ready in the MA system (Fig. 21) and at the same time the ERA team reported that the robot was 
ready, too. At 2:00 PM AstroOne (Brent Garry) said to his personal agent, “Start Egress activity.” 



Figure 21. Day two astronaut EVA plan for Pooh’s Corner. Each icon represents an activity object in the MA 
system, with location and timing information used by agents for alerts, as well as for automatically associating data 
in ScienceOrganizer. 
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All systems go, AstroOne then said to his personal agent, “Start walk to waypoint 12 activity.” After 
several seconds AstroOne’s PA dialog agent said in a new female voice, “Walk to waypoint 12 activity 
started” (refer to Fig. 5). Next, AstroOne said, “Boudreaux, follow me.” Boudreaux’s PA received the 
command a couple of seconds later and forwarded the command to the ERA’s Communication Agent (a 
Brahms agent written in Java), which formulated and sent appropriate instructions to the robot control 
software via the ERA’s API. Boudreaux locked in on AstroOne’s GPS from his backpack and started 
following the astronaut at a safe distance. The EVA had begun. Everything was working, and the 
astronauts proceeded on foot to waypoint 12 at Pooh’s Comer. It had taken Boudreaux almost four hours 
to get there on Tuesday during its autonomous traverse. However, today it was tracking the astronaut in a 
level area, and so we were able to turn off automated obstacle avoidance. The astronauts reached 
waypoint 12 in 30 minutes with Boudreaux following. This 700 m traverse was a record for Boudreaux 
while following a person. 

A potential problem at the beginning of the EVA now became known to AstroOne. Early on, we 
learned in the hab while monitoring all of the agents from the HabCom computer that AstroOne’s GPS 
agent in the Brahms model was not receiving any GPS data from the GPS system. HabCom inferred that 
AstroOne’s PA would probably not be able to download his voice notes. This was confirmed when 
AstroOne finished his first voice note, and HabCom detected that the “speech act” from AstroOne’s PA 
did not arrive at the HabCom PA running on the HabCom computer. Thus, none of AstroOne’s voice 
notes could be stored inside ScienceOrganizer or Compendium. However, AstroOne’s dialogue system 
was recording his voice notes and storing them on the backpack computer, so they could be retrieved after 
the EVA. This combination of backup systems and debugging telemetry was an important part of the 
MAA design for the 2004 field season, based on difficulties observed in previous years. Put another way, 
we were now using the MAA architecture for runtime monitoring, diagnosis, and repair of system 
components during EVAs themselves. 

A second problem occurred when AstroTwo attempted to download all images. HabCom detected, 
again inside the hab using the remote monitoring tools, that AstroTwo’s camera had inexplicably been 
associated with drive E: instead of the normal drive D:, and thus her PA could not find the image 
directory. HabCom was able to remotely change how the camera was on AstroTwo’s backpack computer. 
She repeated the command, “Download all images,” and then less than a minute later HabCom detected 
the pictures being downloaded into ScienceOrganizer in the hab. 

But then the process responsible for communicating with ScienceOrganizer inexplicably quit. This 
meant that the RST would not receive any of the science data during the EVA itself. HabCom detected 
that the RST was online and examining the problem remotely. The problem was resolved half way 
through the EVA; data did enter into the ScienceOrganizer database, and the RST received corresponding 
email (see Fig. 22). 

After almost three hours, the batteries of the backpack computers were almost empty. The astronauts’ 
PAs monitor the battery level and warn the astronauts and HabCom of low battery levels. We hot- 
swapped the batteries, providing another two to three hours. About 5:00 PM, approximately three hours 
since egress, we decided that the crew should start returning to the hab (the EVA was planned, but timing 
was flexible to allow for appropriate diagnostics and testing of the system, with the total time determined 
mainly by the batteries). The EVA crew had completed three of the four sampling activities on the plan 
and asked Boudreaux to take a panorama and a number of images. AstroOne walked over to Boudreaux, 
stood in front of his cameras and spoke again, “Boudreaux, follow me.” Like a trusted mule, it pulled the 
science trailer and followed the astronauts back to the hab. The ingress activity started at around 5:15 PM. 
The EVA times and distances appear in Table 1. 
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Table 1. Distances and duration outside hab during 


Pooh’s 
Corner EVA. 

Robot/Person 

Distance 

Traveled 

(m) 

Duration 

(hr) 

Boudreaux 

1017.2 

2.748 

AstroOne (Brent) 

2745.45 

2.757 

AstroTwo (Abby) 

1983.01 

2.745 


But work was not yet finished for the 
astronauts. After removing suits and a 
debriefing with the MA team, the geologists 
started sample analysis in the lower floor of 
the habitat. The RST had provided a sample 
methodology process that included taking 
photographs of the samples inside the hab and 
providing additional voice notes. The MA 
system was shut down in the backpacks, but 
the HabCom computer stayed operational to 
allow the astronauts to create these voice 
notes and to download pictures from their 
cameras, so that they could be stored and 
associated automatically with other data from 
the EVA. At least this was the plan. 


Unfortunately, when the HabCom computer was brought from the upper deck, the system was unable 
to recover from a broken state. The speech dialogue system became unresponsive, and the crew could add 
no more science data. After dinner, we decided to use Compendium instead. Both geologists used their 
stored and downloaded images and voice notes to provide an analysis of the EVA. Finally, by midnight 
we were able to export the crew’s EVA analysis from Compendium and upload it into ScienceOrganizer, 
just in time for the RST to download it and have their 5 AM SOWG meeting to provide feedback to the 
crew. 
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C. Geologists’ Report on Pooh’s Corner EVA** 

Overall impression: Extremely successful EVA in comparison to last field season in regards to the 
robustness and sophistication of the system. The EVA lasted 4 hours in a very “real-time” and realistic 
atmosphere. The ERA played a prominent and crucial role in helping the astronauts achieve their field 
goals by providing a workstation and reducing fatigue by carrying important equipment. Our 
accomplishments are summarized here. 

1. EVA Plan 

We accomplished all tasks up through “Sample at Red Hill.” Time constraints prevented “Sample at 
Little Red Hill,” “Soil Core” activities, and “Stow equipment.” “Walk back to Hab” was manually 
instigated by HabCom. 

2. ERA and Science Trailer 

• The ERA followed AstroOne to and from the field with no manual control, using the 
“Boudreaux, Follow Me” command. 

• The ERA served as a base of operations for sampling at Rock Hill and Back Hill locations. 

• Science Trailer is useful as a storage tool for samples and equipment. There were six sample 
bags between the two astronauts. Samples and tools would have been too cumbersome to transfer between 
locations in the field and to carry back to the hab at the end of the EVA. 

• Improvised booklet of maps, panoramas, and voice commands was attached to the top of the 
Science Trailer as a field reference. This proved useful when we needed to refer back to new voice 
commands. 

3. AstroOne 

• My first EVA with full human-robot interaction with the robot acting autonomously to my 
commands. 

• Majority of time was spent working with commanding Boudreaux (the ERA) such as follow 
me, take a panorama, take picture of me, and watch me. 

• Was able to sample at Rock Hill and The Mount using the nifty nabber, special tongs to grab the 
rock samples without bending over. 

• Able to create sample bags with the Letter Letter / Digit Digit / Digit Digit format. With this 
format we could indicate sample location, astronaut who took sample, and sample number (e.g. RK/1 1/01 
is for RocK hill taken by AstroOne at station 1 sample 01). 

• Able to lead ERA to the field and return the robot safely. 

4. AstroTwo 

• Managed to follow the majority of the plan (e.g. Egress, Starting Bio-sensors and GPS tracking, 
and starting Sample at Rock Hill). 

• Able to name locations and associate images, voice notes, and samples with locations. 

• Sample bags could be associated with voice notes, however I had problems with associating 
images and voice notes. 

• Sample bags were taken at Rock Hill, Back Hill, and Red Hill with associated voice notes at 
each location and images were associated with each location. 

• Tried out a number of voice commands: What time is it? Where am I? How much time is left?, 
What is current activity? Responses were timely and accurate. 

5. Room for Improvement 

• Astronauts need to stay within safe and helpful distance to each other. This was exhibited by the 
need to help with stowing equipment, also during equipment failures (e.g. AstroOne’s camera batteries 
died [probably from the cold], and needed AstroTwo to take a photo, while at Back Hill). The other 
astronaut serves as a useful scale in photos and to indicate cardinal directions. 

• Push-to-talk (PTT) function on the radios limited the astronauts in discussing what tasks to 
perform and observations and interpretations of the outcrops. A VOX communication system would be 
much more helpful and natural for the astronauts to communicate with each other. The PTT function 


‘ 4 This is a lightly edited first-person account written by the two geologists immediately after the simulated EVA. 
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interrupts making voice notes and is mentally distracting from the task at hand. Finding the button or 
waiting for someone else to finish talking is time consuming. Communication about what tasks the other 
astronaut has accomplished at the outcrop is hindered by different styles of working and lack of 
delegation of who was to perform which tasks. 

6. Post-EVA Analysis 

• Samples were photographed individually in the Flab. The astronauts used compendium to create 
a map of their own descriptions, observations, and inteipretations for each sample and location. This was 
sent to the RST to help them with their analysis. 

VIII. Head of Canyon EVA Scenario Planning 

The second week of the field experiments was devoted to another site, about 5 km from MDRS. Three 
segments were planned for different days, at the head of the canyon (in which the ERA would provide 
surveillance by moving on voice command between predetermined waypoints at the canyon rim), around 
the bend, and down the long winding portion with the ERA (described in the next section). 

This section discusses the EVA planning by the crew of segment one, at the head of Lith Canyon. 
Two planning days were involved (May 3 and 4). This report demonstrates the workflow process and 
collaboration between the crew and the RST over two days. 

A. Crew Initial Planning of Head of Canyon EVA 

In the afternoon of May 3rd, the crew had an hour-long initial EVA planning meeting. In this meeting 
the crew had two objectives: 1) to review the available data for Lith Canyon and based on that define a 
doable EVA plan for next day’s EVA, and 2) to provide the RST with as good a background picture of 
Lith Canyon and the EVA plans in the coming three days as possible. In order to support the second 
objective, the crew decided again to record the meeting on video, so the Meeting Replay tool could be 
used. The crew started the meeting with the creation of an aerial map with and explanation of Lith 
Canyon (See Fig. 23). 

Using this map, the crew explained to the RST how they divided Lith Canyon into three segments, and 
how they planned two EVAs to geologically investigate the three segments. The next day’s EVA would 
cover Segment 1, while the EVA after that would cover both Segment 2 and 3 in one EVA. The scale of 
the aerial image in Figure 1 is 1 meter per pixel making Lith Canyon about 600 m long. 

After the introduction of Lith Canyon to the RST, the crew focused on what they would be doing in 
next day’s EVA. To do this, the crew had created a hand-drawn map of Segment 1, which included hand- 
drawn GPS waypoints for Boudreaux and for the crew to use for demarking the plan. (Fig. 24). 
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Figure 23. Crew annotation of Lith Canyon aerial map. 


i hvri 




>©^i ; 


.‘it 






The crew’s plan was 
to investigate Segment 
1 by entering the 
canyon from Fossil Hill 
(waypoint 5). During 
the investigation of 
Segment 1, Boudreaux 
is to be commanded by 
AstroOne to move to 
waypoints 1 through 3. 
The goals were to 
provide: a) wireless 

communication relay 
back to HabCom while 
at the bottom of the 
canyon, and b) video 
feed and images of the 
astronauts in the 
canyon. To allow for 
total coverage in the 
canyon, the crew 
devised three waypoints 
at the rim of the head of 
waypoint 1 allowed 
Boudreaux to cover the astronauts while 
they are descending into the canyon, as 
well as when they are working on the 
right side of the canyon (see Fig. 23). 
Waypoint 2 allowed Boudreaux to cover 
the astronauts when they are working on 
the left side of the canyon. Waypoint 3 
allowed Boudreaux to visually track the 
astronauts when they are in the center of 
the canyon and moving towards the end 
of Segment 1 (referred to as “the 
Ledge”). The astronauts were able to 
command Boudreaux to move to either of 
these waypoints at will. 

The crew ended the meeting 
discussing how to proceed during the 
Segment 1 EVA. Some of this was 
captured in the Compendium map shown 
in Fig. 25, although the majority of the 
discussion was done using the maps of 
Fig. 23 and 24. It was therefore important 
for the RST to be able to view the 


the canyon. 


Figure 24. Crew hand-drawn map of Segment 1 of Lith Canyon Meeting Replay, 
with waypoints. 
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Where should Boudreaux take Pictures? 


CD 


Start Boudreaux Watch me when 
descending into the canyon starting at 
Fossil Hill 


CD 


You can have Boudreaux take a picture 
of AstroOne at any time alter that 


CD 


Make sure that Boudreaux is in line of 
sight from AstroOne. Thus move it to 
WP 2 and 3 at appropriate times 


Figure 25. Crew’s initial planning map of Head of Canyon EVA using 
Compendium (excerpt). 


B. Crew RST Debrief Review and Final Planning of Head of Canyon EVA 

As planned, the morning of May 4 was used to set up the communication network to Lith Canyon. 
During this time the crew developed the final EVA plan. One of the first things noticed was that the crew 
downlink lead had made a mistake in the filenames of the downlinked video files the night before (the 
RST notified the crew of the problem by email). Consequently, the RST was unable to use the Meeting 
Replay tool during their early-morning meeting. This simple mistake was the result of fatigue, and 
illustrates why such tasks, which are easily automated, should be handled by personal agents. Here we 
were beginning to see how the MAA framework was useful for “office work” in the hab. This example 
illustrates how the empirically grounded development and field-test methodology helps determine 
requirements from a human-centered perspective. 

The RST was able to use the Meeting Replay tool later in the day and reviewed the video of the 
crew’s initial planning session (Fig. 26). 

Meanwhile, the RST had viewed the EVA plan (Fig. 25) in ScienceOrganizer, enabling them to 
provide some timely feedback to the crew for its final EVA planning session on May 4 (Fig. 27). The 
crew incorporated all of the RST’s suggestions in the final EVA plan. 

Weather prevented performing the EVA on May 4 as planned. The RST used the Meeting Replay tool 
and provided further feedback to the crew at the end of the day. The EVA was then run on May 5. 
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On the way into the canyon, AstroTwo made a 
voice note at Fossil Hill, while AstroOne made a 
panorama from canyon floor. ERA successfully 
watched AstroOne for most of the EVA and took 
an image of him. AstroOne worked on the South 
Cliff Face, whilst AstroTwo worked on the East 
Cliff Face, taking voice notes and samples, with 
images. The EVA was ceased due to lack of 
sunlight about 8:30 PM. 


Figure 26. Meeting Replay of Crew’s Head of Canyon 

EVA Planning. Video inset shows the crew inside MDRS 
using Compendium and reviewing the aerial and hand-drawn 
maps to plan the EVA. 
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rstlithsegl crewplandebrief.txt 




1 . Change Where is Pooh's Corner to 
Where is Lithe Canyon (technicallity) 



4. Have ERA take photo of Astro One 
when Astro One takes pan images that 
include locations that will 


Figure 27. RST email feedback brought into Compendium by the crew. 


The following section provides the Commander’s view of the subsequent day, preparing for the Segment 
3 EVA on May 7. 


IX. Preparations for Lith Canyon EVA §§ 

Log Book for May 6, 2004 
Commander's Check-In 

Time: started 8:19 AM May 7, completed 9:51 AM 

Weather: low 15c (59F) high of 29.1c (84. 4F), hot and dry (12% humidity) with mid-afternoon winds 
Crew Physical Status: Very good, one person with minor scrapes. 


This is an edited version of the report submitted by the expedition leader. 
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Narrative of Field Mission Results 

Today, May 6, was another successful day. We began with a briefing at 9 AM, joined by a NASA 
Ames management observer. 

Preparing for the day’s test required a number of steps, which would take most of the day: 

1. To deal with the overloading of the ERA’s computers, we would use the spare laptop for 
running the Brahms-ERA agent system, locating it in the rental truck at Lith. This computer would be 
virtually “on board,” communicating wirelessly with ERA systems. Configuring the machine and relating 
it properly to the ERA might seem complicated, but the modularity of our systems made it relatively easy. 

2. To handle the overheating of the astronaut computers, we would reconfigure the ventilation in 
the metal-enclosed backpacks, adding small fans right against the CPU area, with an additional opening 
on one of the backpacks. We would use a flow-through method, rather than just pushing or pulling the air. 
The metal cutting and electrical work required a good part of the day. 

3. Once the first two steps were completed, we’d do a fiill-up start procedure in place at Lith, 
without intending to run a simulated EVA. We would verify the sequence of starting we had identified 
yesterday (roughly: network, RIAL1ST, Mobile Agents, biosensors, and GPS). We've learned that the 
sequence of plugging in the headsets, sensors, and camera is crucial; unplugging after the system starts 
can cause odd behavior later. 

4. We would examine the far point Cisco repeater (.5 km from our EVA site) and recharge the 
battery if necessary. 

5. We’d replace the push-to-talk mechanism used by AstroOne. 

I retired to my stateroom-office to write the previous day’s report and process photographs. I also 
greeted a Spanish-speaking TV group from Salt Lake City and gave an orientation to the Ames visitor. 
These activities filled my morning, then I turned to email, including responding to more than 20 fairly 
interesting questions from an Italian reporter. 

The winds picked up at the canyon, requiring equipment to be moved into the truck and the canopy to 
be tied down even more securely. But the network backbone held with good throughput to the hab. We 
decided to go out about 4 PM after testing showed that the fans allowed the backpack computers to run 
cooler than those just sitting on tables in the open truck. An ERA team member had carried around the 
backpack doing speech tests with the ERA, which showed all was in order. We decided to do a short 
EVA, rerunning the head of canyon scenario. 

What followed was our quickest start to date: We arrived at Lith at 4:40 PM, started the Mobile 
Agents system from 5-5:13 PM, changed to internal battery at 5:18 PM. By 5:30 PM AstroTwo was fully 
outfitted and starting her sensor systems by voice co mm and. The EVA then proceeded in its usual chaotic 
way, lasting nearly 1.5 hours. 

Conditions were excellent for this EVA test — moderate to no wind, temperatures in low 80s, partly 
cloudy, beautiful late afternoon lighting. Of course not everything went according to plan. 

The ERA had some GPS trouble and did not ever reach a waypoint by voice command (though it came 
close on the last try, losing GPS during the operation). The robot was successfully teleoperated from the 
hab to provide video of the work in the canyon. The astronauts commanded the robot to take photos and a 
panorama from the ledge, looking down at them on the canyon floor. (However, one photo was a bit late, 
and AstroTwo had already moved out of sight.) 

For about an hour, the geologists went about their work of naming work locations, taking photographs 
and samples, and making voice notes. New commands like “Associate last voice note with last sample 
bag” and automatic naming of downloaded images as “collections” occurred properly. By using location 
names that begin “Work Site” (e.g., WorkSite5), the system automatically associated subsequent sample 
bags and voice notes with that location (so additional commands are no longer required). The data were 
properly stored in the ScienceOrganizer database in the hab (transferred automatically by Mobile Agents 
from the HabCom Brahm s system) and copied by MySQL to the mirrored database at NASA Ames. 

Members of the RST received email of each filing in ScienceOrganizer (similar to Fig. 8). Notice how 
the EVA plan, activity (established by the astronaut during each phase of the EVA), astronaut name, exact 
location and time are stored. 
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As a member of the hab crew, I received copies of each message, so I can quickly review the sequence 
of what occurred during the EVA from my mail folder. 1 see that AstroTwo took 19 photos, used four 
sample bags, and made three voice notes. She named three work sites, spaced 20 and 16 minutes apart. 
An email even indicates the moment when her GPS signal was lost (as she stepped under an overhang at 
the lip of the canyon) — a milestone for this test, because her backpack computer remained on the 
network, with the ERA serving as a relay to the backbone. Various alerts about heart rate and Oxygen 
saturation were also sent, but it’s not clear whether these sensor readings were correct. 

Although we were disappointed that the ERA could not be moved from side to side by voice command 
to keep the astronauts within camera view, this was actually an improvised addition to the scenario last 
week. We in fact met our objectives for this segment of establishing the ERA as a relay and using the 
Mobile Agents system to properly document the geology of an area. 

In comparison, last year’s test at Lith lasted only about 20 minutes and only proved that the basic data 
logging commands worked; only one image was actually received remotely. This year we used the system 
for over an hour in the canyon, used much more sophisticated naming and association automation, and all 
of the data were stored automatically in the hab and a remote database. 

We have clearly turned the comer from just getting the system to work at all, to having a system that 
can actually be used experimentally and improved by observing how people use it. 

We arrived back in the hab before 8 PM. We were treated to a pizza dinner by NREN, including the 
entire expedition team (with visitors from two communication network vendors). It was a fun and 
relaxing evening. Our celebration was tempered only by the absence of the support person who had set up 
the party, now hospitalized in Provo with a heart problem (though doing well). 

The intensity and excitement of the day’s work continued late into the evening, as we watched a 
“video file” on NASA TV after 11 PM. The 17 minute segment, filmed and edited by a team at NASA 
Ames, showed our work last week and included interviews with the crew. It was eerie to watch this video 
sitting in bed in my stateroom, using my laptop on the wireless network. Others watched on the big screen 
in the main room of the upper deck. 

With the draw on the network, the image was out of synch with our voices. And bizarrely, adding to 
the sense of relativity, my stateroom connection was about 20 seconds ahead of the wardroom’s, so I 
provided narration about what was coming next. Our experience was a hall of mirrors, of present and 
future echoing and twined together-just like our field tests, both physically real here and now, yet in our 
imaginations and videos, projecting the reality of people on Mars. 

Plan for Tomorrow: Do Segment 3, in which the ERA follows the astronauts in the long run out of 
the canyon towards the east and north. 

X. Long Relay EVA Scenario (Lith Canyon) 

This section provides reports from the ERA team and the geologist-astronauts on the May 7 EVA into 
Lith Canyon (Fig. 23). 

A, ERA Team Report 

May 7 was our one and only attempt for Segment 3 — an endurance challenge designed to run-out 
communications, data, batteries, and Boudreaux. 

Segment 3 required the ERA team to belay Boudreaux down to the bottom of Lith Canyon. There 
Boudreaux towed the science trailer carrying tools and equipment to assist the astronauts as they stressed 
the limits of the wireless data network, the mobile agents infrastructure, and their own stamina in the suits 
and the heat. 

• The ERA team left the motel at 7:30 AM to give plenty of time to belay Boudreaux the robot and 
its science trailer approximately 75 m along a sloped path down into Lith Canyon. Five men were 
required to perform this operation. 


This is an edited version of the report submitted by the ERA team after the May 7 EVA. 
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• The EVA simulation started at approximately 20:10 UTC (2:10 PM local time). 

• The obstacle avoidance system came very close to failing once at 22:25 UTC, slightly scraping a 
large rock after two forward-backward-tum attempts around it. An e-stop was pressed by field 
support crew, but was strictly unnecessary. AstroOne was asked to issue a “Halt” voice command 
to reinitialize the rover (which was slightly odd from a human perspective, given that the rover was 
obviously stopped.) 

• ERA’s computer inside AstroOne’ s infoPak backpack reached a peak CPU temperature of 44 C 
(111 F) and never failed due to heat. The MA computer inside the same infoPak did fail due to heat 
(something above 55 C (131 F). CPU loading on the MA computer and poor venting at the laptop 
caused the high heat and failure (ambient temperature was 83 F with 7% humidity and no wind). 

• The simulated EVA ended at approximately 23:28 UTC. 

• Boudreaux suffered no hardware failures or anomalies today, and required no primary battery 
swaps. The ERA laptops had their batteries replaced at the start of the EVA start (they were up and 
running since 17:00 UTC, when the EVA was originally scheduled to begin). After the run was 
over, one of the main laptops onboard Boudreaux failed (dead batteries) because it was not 
immediately removed and placed on AC power. 

• The following objectives were accomplished today from the ERA’s perspective: 

• Six “take image” commands 

• Four “take panorama” commands 

• Six “follow me” commands 

• Five “watch me” commands 

• The interesting combination of watching one astronaut with a pan-tilt camera while following 
the other astronaut. 

• Four “stop” commands 

• Multiple “report position” commands 

• Multiple “define position” commands 

• One “report ground-track map” command 

• One “report elevation map” command (after a fixed bug on a failed first attempt). The map 
itself was not very useful since AstroTwo was not reporting elevation due to using only absolute 
GPS hardware. 

• Speech output from Boudreaux through Brahm s out to the astronauts and HabCom (using new 
server and listener software). 

• Situational awareness of the remote EVA from HabCom inside MDRS was achieved by 
providing live imagery from Boudreaux and various GUIs, some showing the agent positions 
and their movements. 

• Boudreaux served as a data communications relay from the astronauts up to the repeaters (ATV 
and antenna on ridge, see Fig. 2) and back to the Hab. 

• A series of still images (1 fps) were collected of the entire EVA to be stitched together as a short 
movie. 

• Only one objective was not accomplished today: the “print curation label” command failed due to a 
hardware failure in the printer itself (probably due to sand in the mechanism). 

• The total times and distances traveled today were: 

• Boudreaux, 3.3006 hours, 289.937 meters 

• AstroOne, 3.2945 hours, 1740.931 meters 

• AstroTwo, 3.2882 hours, 1517.784 meters 

• After the run, five men again belayed the robot and science trailer up from the canyon floor and helped 
it get back to the camp. 

• The ERA team packed up and secured everything for the ride along rough BLM roads from the Lith 
site back to the MDRS Habitat and the motel for packing. 

To summarize the day, all field teams had an excellent run, and the ERA team performed well. The 
Lith Canyon challenge has finally been met, though not conquered. 
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B. Geologists’ Report on Lith Canyon EVA ttt 
Lith Canyon had been separated into three segments, planned for three sequential days (Fig. 23). Due to numerous 
repetitions of Segment 1 to get all of the equipment functioning, Segment 2 was skipped in preference of Segment 3, 
which incorporated the ERA and gave more opportunity for geology. Segment 3 is summarized here. 

• Morning was spent creating a plan for Segment 3. This was difficult due to lack of images for this area - 
we decided to employ a basic “walk to the canyon, move along the segment and create worksites at areas 
we found interesting geologically.” 

• The astronauts were on the canyon floor by 2:20 PM (local MST) and the EVA ceased at 6: 10 PM - very 
successful EVA in all! 

• Creation of worksites and the subsequent association of voice notes, samples and images all worked well. 

• ERA followed AstroOne for the majority of the Canyon. 

• Astronauts were able to lead the ERA trough a complicated “S-turn” in the canyon. 

• ERA’s panoramas and images were commanded by AstroOne primarily with some by AstroTwo to 
demonstrate that either astronaut could command the robot. 

• Astronauts managed a good EVA with one battery change and a refill of water. Models only failed at the 
very end of EVA; AstroTwo ’s model was shut down accidentally by HabCom [which indicated an 
opportunity for intervention by his personal agent!]. 

• Numerous worksites, samples, images and voice notes were created throughout the EVA. 

• Geologically, the astronaut’s walked “down-section” into older layers of rock than were found at the head 
of the Canyon. Exposure of fresh faces were more abundant in Segment 3. 

• We were very tired at the end! 

XI. Results and Discussion 

The Mobile Agents 2004 experiments demonstrated that the system has transitioned from an 
engineering test phase to experimental use doing authentic work, in particular geologic exploration in 
terrain that was new for the participating scientists. Different teams analyzed the data appropriately for 
their needs, focusing on performance of the robot (ERA), voice commanding (R1AL1ST), 
communications network (MEX), agent processing (Brahms), and work system design (i.e., correctness, 
efficiency, and naturalness of the overall tool-using experience relative to the work to be done in the 
canyon and habitat setting). 

Analyses are primarily statistical for each category, using history data recorded during the various 
scenarios. For example, a complete log exists for all Brahms agent state changes, including changes in 
belief, activities, and primitive actions (including communications between agents). This log may be 
viewed graphically, such that one may inspect at any moment what each agent on each platform was 
doing and how they were interacting. Similar information exists for the ERA, RIAL1ST, and MEX. An 
excerpt from the Lith Canyon (May 7) EVA is analyzed in the next section. The subsection following 
details many improvements based on the 2004 experiments. 

A. Example analysis of voice commanding interaction 

Of particular value is the complete record of speech from the astronauts and ElabCom, as well as a 
record of what speech was generated by agents as questions or feedback. This data can be consolidated 
into a table, allowing the full chronology of interactions to be grasped (Table 2). The time is local MST. 
The astronaut designates one of the EVA crew members or ElabCom. The origin indicates either a voice 
recording by an astronaut (.wav file) or the personal agent of an astronaut or the ERA (in some cases, it is 
actually the corresponding dialog agent that generates the speech act). Finally, the actual speech spoken 
by the astronauts or generated by the agent is given. When no speech appears, the astronaut’s remark was 
not interpreted to be a voice command (e.g., it might be a conversation with the other astronaut or an 
external support person). 


This is an edited version of the report submitted by the geologists after the Lith Canyon EVA of May 7. 
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Table 2. Excerpt of voice interactions during May 7 Lith Canyon Scenario. 


TIME 

ASTRO 

ORIGIN 

SPEECH 

23:38:46 

1 

utt103.wav 

Boudreaux stop 

23:38:46 

1 

PA1 

Do you want Boudreaux to halt? 

23:38:51 

1 

utt104.wav 

yes 

23:38:51 

1 

PA1 

Asking Boudreaux to halt. 

23:38:53 

2 

utt408.wav 


23:39:01 

2 

utt409.wav 


23:39:08 

2 

utt410.wav 

take a voice note 

23:39:08 

2 

PA2 

Do you want to record a voice note? 

23:39:12 

2 

utt41 1 .wav 

yes 

23:39:12 

2 

PA2 

Please begin recording voice note now: 

23:39:16 

1 

PA ERA 

Tracking sequence aborted 

23:39:18 

habcom 

PA ERA 

Tracking sequence aborted 

23:39:18 

2 

PA ERA 

Tracking sequence aborted 

23:39:32 

1 

utt105.wav 


23:39:41 

1 

utt106.wav 

Boudreaux take panorama 

23:39:49 

1 

PA 1 

1 am still waiting for a response on your command 

23:40:15 

1 

utt107.wav 

Boudreaux take a panorama 

23:40:16 

1 

PA 1 

Boudreaux stopped. 

23:40:16 

1 

PA 1 

Shall 1 take a panoramic image named panoramic image 1 here? 

23:40:41 

2 

PA 2 

Finished recording voice note 

23:40:41 

2 

voice_note_ 

_2004-4-7_23-39-8.wav 

23:40:44 

2 

PA 2 

Voice annotation is created 

Do you want to associate Astronaut Two Work At Way Point16Work Site7 

23:40:44 

2 

PA 2 

with Astronaut Two Model VOICE ANNOTATION 7? 

23:40:48 

1 

utt108.wav 

yes 

23:40:48 

1 

PA 1 

Taking a panoramic image named panoramic image 1. 

23:40:58 

2 

utt413.wav 

yes 

Attempting to associate Astronaut Two Work At Way Point16Work Site7 with 

23:40:58 

2 

PA 2 

Astronaut Two Model VOICE ANNOTATION 7. 

Voice annotation voice note 8 is associated with location Astronaut Two 

23:41:01 

2 

PA 2 

Work At Way Pointl 6Work Site7. 

23:41:24 

2 

utt414.wav 

create sample bag echo lima slash two seven slash zero one 

23:41:24 

2 

PA 2 

Do you want to create sample bag echo lima / 2 7 / 0 1? 

23:41:31 

2 

utt415.wav 

yes 

23:41:31 

2 

PA 2 

Attempting to create sample bag echo lima / 2 7/01. 

23:41:34 

1 

PA 1 

1 am still waiting for a response on your command 

23:41:34 

2 

PA 2 

Sample Bag is created 

Do you want to associate Astronaut Two Work At Way Pointl 6Work Site7 

23:41:34 

2 

PA 2 

with sample bag echo lima / 2 7 / 0 1 ? 

23:41:50 

2 

utt416.wav 

yes 

Attempting to associate Astronaut Two Work At Way Pointl 6Work Site7 with 

23:41:51 

2 

PA 2 

sample bag echo lima / 2 7 / 0 1 . 

Sample bag echo lima / 2 7/01 is associated with location Astronaut Two 

23:41:54 

2 

PA 2 

Work At Way Pointl 6Work Site7. 
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Several interactions are noteworthy, illustrating how the analysis process proceeds: 

o It can be seen that Boudreaux required nearly 1.5 minutes to halt. On observing this, and 
anticipating the safety problems it might cause, we have decided that confirmation of this 
command should occur after it is executed, rather than before, 
o Notice that when Boudreaux halts, all members of the crew are notified that the “tracking 
sequence is aborted.” The synthetic voice indicates that it is “spoken” by the ERA. This might 
be distracting to AstroTwo, who has just begun recording a voice note, and is not directly 
working with the ERA at this time. Hence, context might be taken into account in determining 
who receives this feedback information. 

o AstroOne’s sequential statements at 39:32 and 39:41 suggest that possibly uttl05.wav was a 
command that was not understood. We can listen to the file to determine the cause of the 
problem, and indeed use the recording to refine the speech recognition system, 
o AstroOne asks the ERA to take a panorama before the stop acknowledgment has been 
received. Before this command is processed, PA 1 tells the astronaut that it is still waiting for 
a response. But it may not be clear which command it is referring to at this point. It would 
have been clearer if PA 1 said, “I am still waiting for a response on your halt command.” 
o Meanwhile, AstroTwo has been recording a voice note, which lasts for about 1.5 minutes and 
is stored in the file indicated at 23:40:41. This interaction, like the halt and panorama 
commands from AstroOne are being relayed back to the HabCom computer, with 
corresponding data stored in ScienceOrganizer. The agent confirm s that a database link should 
be created that associates the voice note with the astronaut’s activity (“work at waypoint 16”) 
and the work site (number 7) she named previously, during this EVA. (However, the feedback 
at 23 : 41:01 indicates an error, in referring to voice note 8 .) 
o A similar interaction follows, in creating a sample bag with a name improvised on the spot. 
AstroOne is told at this point that the ERA’s panorama to not yet complete. When actual 
problems develop here, the HabCom should have been notified with an alert, 
o Keep in mind that while all of this is occurring, agents are simultaneously tracking biosensors, 
location, and duration within the activity, generating appropriate alerts when thresholds are 
exceeded. 

B. Summary of Discoveries and Tentative Design Recommendations 

This subsection summarizes by component category the most salient observations made from the 2004 
experiments and design changes that are being considered: 

1. ERA Use during EVA 

o Tele-operate ERA (using video and pancam) to allow the crew to remotely explore new region and 
define work sites (with dGPS locations), avoiding physical walkthrough prior to an EVA. 
o Clarify why using the ERA with a single shared trailer in an area such as Lith Canyon is 
superior to using individual ATVs for the astronauts with trailers and independent video, 
o Investigate ways to increase speed while using obstacle avoidance (a problem of gathering 
sufficient data); tune motor controllers; use more powerful motors; add remote kill capability, 
o Add an arm and hand to each ERA 
o Build 3D panoramas 

o Add voice commands to enable astronauts to obstacle avoidance on turn and off (tests showed 
that it is unnecessary when following astronauts in relatively flat, unobstructed terrain), 
o Train astronauts using voice commanding and handling the robots more extensively prior to the 
field experiments (this is usually a matter of timing and availability, as the astronauts are 
students in NY). 

o Add/Improve robot speech so Astronauts have better awareness of robot activity. Specific 
examples: ongoing updates on tracking, taking panoramas to reassure astronauts that robot is 
healthy, and obeying commands. 

o Make the ERA PA more proactive in monitoring safety and ease of use: 
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• During Voice Note recording, ERA automatically takes photograph of the astronaut to 
establish the setting 

• ERA automatically follows astronauts and watches them when they are moving to the next 
activity (ask for confirmation) 

• “Halt" is executed first, then confirmed; continue moving otherwise 

• Monitor astronaut location with respect to communications relay (maintain line of sight with both 
astronauts and ensure that they can see each other) 

• Do image processing to detect an obscured astronaut or one in medical trouble (combining with 
biosensor data and activity expectations, e.g., when is lying prone on the ground acceptable?). 

• Reject move commands that will break communications relay 

• Detect and locate objects by gestures, e.g., “Take a picture of that rock.” 
o Use a second ERA robot and/or a Athena (MER-class) robot during EVAs: 

• Leave a robot behind to do analysis for reporting during the EVA, providing information 
useful when the astronauts return later to that spot. 

• Use a second robot as a communications relay, moving the two robots in a teamwork 
manner to accomplish better coverage 

• Teleoperate a robot from the hab (or via program uploads from “Earth”), using it for 
reconnaissance of potential EVA sites or revising places identified by the astronauts as 
worthy of detailed investigation. 

2. Agent Language Improvements 

These suggestions are based on interactions between the astronauts the support personnel, indicating 
communications that should be had with their personal agents: 

o Provide information about the robot’s state, e.g., “Boudreaux, whom are you following now?” 
o Provide information for navigation, scheduling, and work activities: 

• Where is {waypoint # | the next waypoint} ? 

• Where is the next activity? 

• How do I get to location X? (X = name in the plan) 

• Where is astronaut N? 

• What is astronaut N doing? 

• When do I/we need to turn back? 

o Information about EVA histories, e.g., “Where did I/we sample X previously?” 
o Enable astronauts to classify samples using descriptive language 
We also need to properly implement “generic” activities based on EVA ontology: 

{Move, Sample/Survey/Document (photo + voice note), Apply Instrument} at <loc> for <duration> with 
<astronauts, robots> 

3. EVA Planning and Data Display 

These suggestions concern the collaboration tools used by the crew and the RST : 

o RSS-style (Really Simple Syndication) EVA data display during the EVA to which RST and support 
teams can subscribe, complementing or replacing emails. 

• Summary and highlights of the EVA to a given point 

• Resource statistics (time, distance), alerts, key URLS 

o EVA “replay” tool - synchronized playback of astronaut and agent voices (now recorded separately 
in different folders for each speaker), plus agent activities (workframe state), showing alerts, 
movements 

o Above to be integrated with Brahms Virtual Environment 40 , a graphic-animation of the EVA for 
viewing state and accessing science data. 

o Personal agent to help analyze EVA data in the hab (“CEV-style downlink agent”), including post- 
EVA curating 

o Better integration of ScienceOrganizer, Compendium, route planning and mapping software, with 
data analysis tools (e.g., Excel spreadsheets) 

o Develop a new planning tool with a customized planning user interface on top of Compendium to 
allow for more structured planning and simplified information entry and validation. The planning 
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tool can be used during an EVA to visualize the status of the plan by interpreting events 
communicated by the agents. 

4. Summary > of significant configuration requirements for the MAA 

o Improved plug-n-play of MA platforms, e.g., carry on an astronaut EVA without the ERA or 
dynamically add a robotic module during an EVA 

o Move plan management from HabCom agent to astronaut PAs, improving response time and safety 
in the event of a communications failure, plus allowing each person and robot to have an independent 
(or specialized) plan and to be able to be run independently of the HabCom computer, 
o Separate ScienceOrganizer Agent from PAs on each platform for efficiency. 

o Eliminate requirement for a second laptop running Linux in backpack for dGPS (by porting software 
to Windows); will reduce chance of overheating in the backpack, 
o Move laptop that manages agent registry from ATV serving as a relay to ERA trailer for more 
realistic configuration, including better directory service and an agent search function, 
o 400 MHz walkie-talkies for all participants to eliminate separate system for astronaut 
communications. 

o Enhanced system monitoring that includes CPU temperature monitoring to prevent unexpected 
computer failures. 

o Develop a procedural agent that relates astronaut actions to semantics of generic activities, to allow 
CapCom-style monitoring of what the astronauts are doing at each work site, e.g., alert if did not take 
a sample during a site survey activity. 

o Develop a medical agent for more systematic health monitoring and alerting of crew members during 
an EVA. 

o Develop a navigation agent that reorganizes functions currently in the HabCom PA and has improved 
capabilities to respond to information requests and commands (listed above), 
o Similarly, reorganize data storage and proxy agent functions for increased efficiency and modularity, 
o Store and make available EVA data to the agent system over long durations (starting with days and 
continuing to years). Reminding and alerting functions can be augmented to search and relate current 
activities to past data, methods, difficulties, etc. 

o Improve agent models overall to retain and work within activity contexts, rather than being strictly 
reactive (speech-act driven). 

We believe that the increasing number of ideas generated from field experiments each year indicates 
the rich potential of our approach, involving incremental development of a useful tool, within an authentic 
work context, with broad involvement of many roles in integrated simulations. The Conclusions section 
comments on how this work fits within the overall objectives of planetary EVAs and how the sequence of 
Mobile Agents experiments exemplifies an analog missions field campaign. 

XII. Conclusions 

The Mobile Agents Architecture is now ready for development in space exploration missions to 
support a network of coordinated operations between astronauts and remote support teams, robotic 
systems, other devices and systems, and software tools. The key aspects of the architecture are: multi- 
layer communication network relating wireless services to agent software; distributed computers, robotic 
devices, sensors, and instruments; workflow agents (including monitoring, diagnosis, advising, and data 
management); a commanding and feedback protocol; semantic database with automated, context-based 
creation of associations; activity-based analysis and design; tools to facilitate remote collaboration. At the 
same time, the use of speech recognition, GPS, a wheeled robot, etc. are important in advancing the state 
of the art, but not essential elements of the computing methods and architecture. In practice, the ERA 
might be replaced by a personal ATV on which an astronaut could ride, perhaps with tracks rather than 
wheels; a touch screen on a spacesuit cuff could complement or replace speech processing; GPS might be 
replaced by any number of alternative location and navigation technologies. The essential idea is gluing 
together a diverse, distributed set of devices, software, and players by using a multi-agent system to 
monitor and manage workflow. 

Although we believe that the MAA system has reached a state of practical development, it should 
be emphasized that the vision of automating CapCom requires many more years of work. Agents that do 
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health monitoring are only proof of concept, lacking most of the functionality required. The foundation is 
laid for navigation by allowing astronauts to name places, but nothing has been implemented for helping 
the astronauts identify where they are or how to get to other places. Many functions have not even been 
prototyped, such as replanning EVAs dynamically with teams of people and robots, and handling 
emergencies. Although the current voice commanding is impressively robust and useful, the dialog 
system has no capability for real conversational, mixed-initiative interaction, which the state of the art in 
computational linguistics allows. Within each of these arenas, points of flexibility need to be identified 
(as outlined in the first paragraph of this section), such that trade studies with competing approaches and 
technologies can be carried out. 

Referring to CapCom’s role in Apollo lunar traverses, one of the most demanding activities involves 
advising the astronauts about when to move along to the next site, when to take extra time (and what to 
do), and when to skip part of the EVA plan. CapCom provided this advice by interacting with the science 
backroom (in a room located near to the Mission Control Center). For extended lunar missions most of 
the scientists will be distributed in their home locations, as they are today in the extended MER mission. 
Furthermore, for Mars missions and beyond, the time delay will prevent sufficient awareness and time to 
advise work that will occur within 15 to 50 minutes roughly, allowing time to receive data, inteipret it, 
and to reach a consensus opinion. Thus, as many people have noted, 41 EVA advising from Earth will be 
more strategic, considering steps in the current EVA and of course planning work for future days. The 
role of the PAs on the moon or Mars will accordingly require sending important information during 
EVAs to key personnel, relating current results to the objectives of the EVA (e.g., to detect when results 
are disappointing or not significant and allow for cutting time short), and revising the plan being followed 
inside the MA system according to information being received from remote teams during the EVA. One 
idea is that scientists on Earth could have personal agents that themselves “come alive” at appropriate 
moments during an EVA, serving as proxies that inform or remind astronauts of interests as opportunities 
as they occur (e.g., “AstroOne, that rock appears to contain goethite, remember that Dr. Z asked for an 
underlying soil sample when you found goethite ...”). Developing such capabilities in a robust way 
approach some of the classic artificial intelligence problems of modeling new phenomena on the fly. 

In the convergence of new exploration missions (e.g., the Crew Exploration Vehicle) and maturing 
technologies such as Mobile Agents, concern increases that research activities should “flow down” from 
requirements. Although returns on investment are important in the long-run, delivering according to 
specifications is not how research and invention occur. In practice, scientific investigation — which is 
required to properly relate people and robots in extreme environments — develops from existing 
instruments, methods, scenarios, settings, and questions. This repertoire of tools and capabilities can be 
redirected to meet mission objectives, but always some freedom for exploration — here in the scientific 
domains of cognitive science, anthropology, and sociology — is just as necessary as it is for the geologists 
and biologists who investigate the universe. 

In practice, a “middle out” approach, rather than a strict “flow down” from requirements is possible 
and advantageous. Scientists already engaged in experimental work can become engaged in near-term 
mission needs (i.e., for use within 5 to 10 years) by finding connections between advanced approaches 
(such as Mobile Agents) and operations requirements. To a large degree, existing research provides 
essential operations concepts that would not be considered if one were only specifying systems according 
to historical metaphors and tools currently in use. 

For example, consider a Mars scenario with a crew of four in a surface habitat, a MER-like robot 
regularly inspecting external systems, robots like the ERA, and EVAs for two crew members of varying 
durations and distances from the habitat. To what extent can the crew inside the habitat work unimpeded 
on other tasks, such as writing reports, analyzing data, planning future work, repairing equipment, etc.? If 
one starts with historical concepts based on the Space Shuttle, one would assume that one crew member 
will be dedicated to monitoring the EVA for its full duration, maintaining a continuous conversation with 
the crew and guiding them. But Mobile Agents results suggest that a variable degree of attention is 
possible, such that responsible crew member (whom we have called HabCom) receives status information 
that varies from continuous notification (e.g., new activities started by any person or robot) to emergency 
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only alerting (e.g., an astronaut health problem, equipment malfunction, or a violation of a plan constraint 
that is not resolved by the crew within a certain time period). We know generally how to build such a 
system today using the MAA. The questions to be determined are: How safe and practical is emergency 
only alerting (e.g., do the astronauts need to consult with HabCom for many routine problems)? Given 
the loss of situation awareness by not paying attention, and the possible difficulties of reengaging in what 
is happening during the EVA, is there an optimal kind of notification that facilitates handling emergencies 
or even detecting and handling limitations in the agent system? To answer such questions, we could 
develop a Mars analog simulation (such as at MDRS), include an Earth-based safety monitor role, and 
experiment with different designs. This example shows that even such a basic question as whether a crew 
member must always attend to an EVA is called into question by new technology. The implications 
potentially and dramatically change how crew activities, monitoring tools, and software throughout the 
system is conceived and built. 

In the near term, Mobile Agents experiments will shift from running the system until it fails or the 
astronauts are exhausted, to full circuit simulations that monitor resources and advise about turn around 
time and attempt to extend explored areas through EVAs on different days. Thus, schedule priorities, 
teamwork, and well-timed coordination with the RST will become important. In this way, the system will 
incrementally become more and more like Apollo’s CapCom in its capabilities. Following a human- 
centered approach, we will continue to emphasize how people and our computer systems differ, 15 ’ 16 ' 42 thus 
highlighting what tasks must be monitored or carried out by the astronauts or HabCom. We believe that 
this is the only way to reliably, realistically, and efficiently produce human-robotic systems — developing 
on a foundation of what is useful, remaining cognizant of what is possible using the technology, and yet 
have our eye on a vision that stretches technical capability and extends the reach of space exploration. 
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